In addition to the index.ts/files-in-root workarounds, if I add a dependency they are not getting deployed in to the lambda. To workaround that, I have to use cloud-aws:functionIncludePackages: handy-redis or similar.

Is there any support for yarn workspaces ? I think I am facing the same issue on a big monorepo managed using yarn workspaces. Some dependencies seem to not be found by pulumi cli but they are actually located in the node_modules found in the root directory.

Could not include required dependency '@outmind/lambdas' in '/.../outmind-app/aws-architecture'.
Could not include required dependency '@pulumi/pulumi' in '/.../outmind-app/aws-architecture'.
Could not include required dependency '@pulumi/aws' in '/.../outmind-app/aws-architecture'.
Could not include required dependency '@pulumi/awsx' in '/.../outmind-app/aws-architecture'.

All my packages are named @outmind/xxx and here the @outmind/lambdas package has not been found. Strangely, the lambda is successfully deployed but the generated code is pure gibberish.

Thank you for any support. Our company is really excited about this project but we can't manage to make it work in our environment.

Following this as well, we've had in other solutions to really work hard to get around this. Having first
class support for this would be a huge win over many other platform.

I am facing the same issue with lerna/yarn workspaces. The error message is also misleading imo: I do not think we want to include @pulumi/pulumi in the lambda bundle (and probably pulumi doesn't).

I think you should just go to packages/cloud and run yarn add ... with all pulimi packages that you are going to use in that directory instead of just hoping that it will be requires from root

Looked into this a little today.

It appears the issue is that we use https://github.com/npm/read-package-tree to read the local package.json and recursively walk dependencies and their package.jsons to find the transitive dependencies of a package. However, read-pacakge-tree does not handle Yarn Workspaces. It assumes that wherever the package.json file is, the node_modules next to it will contain all of the child modules that are direct dependencies of that package.json.

https://github.com/npm/read-package-tree/blob/master/rpt.js#L109

We will likely either need to maintain a fork of read-package-tree that handles this case, or abandon read-package-tree and find a way to construct these dependencies more directly.

Related: #4349

Facing the same issue here in a yarn workspaces + lerna monorepo. I have a common package where I have shared code. I'm getting the same Could not include required dependency '@project/common' error. Is there plans to add support for this?

FWIW this is my solution. I have created a pulumi.js node script which runs tsc -b and also added dynamoDB locking.

https://gist.github.com/JakeGinnivan/a8a9487b427755f267ea6bf7ec42699f

I just make sure I have a single pulumi project for my mono repo and use webpack to bundle lambdas and code which pulumi uploads

I had success by adding into my pulumi files package.json:

  "workspaces": {
    "nohoist": [
      "@pulumi/aws"
    ]
  },

See here: https://classic.yarnpkg.com/blog/2018/02/15/nohoist/
This basically forces yarn to put some packages locally instead of at the root of the project

@vvo this only works for Yarn v1. This option is gone for Yarn v2.

Has there been any movement on this issue?

As it stands it does not seem like pulumi works with yarn workspaces.

@JakeGinnivan do you have a working example of how you do that?

@DanNeish I've now switched my approach and am building out an NX plugin to make it easy to use Pulumi in an nx.dev mono repo.

I am now trying to create a small pulumi program for each service in the mono repo, then working towards being able to nx affected:up to deploy the appropriate stacks which have changes using NX's impact analysis.

So yarn recently added the nmHoistingLimits option, this ensures that the workspace has a local node_modules.

By adding the following to the package.json in the workspace that is deploying the inline lamdas you can enable this per-workspace:

  "installConfig": {
    "hoistingLimits": "workspaces"
  }

Note: you may have to clear your yarn cache.

The problem I'm facing now is that the way Yarn does this for all local workspaces is to add a symlink.

ls -lart node_modules/@tetrateio-components
total 0
lrwxr-xr-x   1 liamwhite   42 Oct 10 11:12 aws-lambda -> ../../../../../../../components/aws/lambda
...

When Pulumi tries to bundle these it gets confused and thinks this is a file. So throws the following error:

Diagnostics:
  pulumi:pulumi:Stack (test-organization.test.managementplane.api.azure-alpha):
    error: Error: failed to register new resource azure-alpha-delete-tenants [aws:lambda/function:Function]: 2 UNKNOWN: failed to compute asset hash: asset path 'node_modules/@tetrateio-components/aws-aurora' is a directory; try using an archive
        at Object.registerResource (/Users/liamwhite/src/github.com/tetrateio/tetrate/cloud/node_modules/@pulumi/aws/node_modules/@pulumi/pulumi/runtime/resource.js:219:27)
        at new Resource (/Users/liamwhite/src/github.com/tetrateio/tetrate/cloud/node_modules/@pulumi/aws/node_modules/@pulumi/pulumi/resource.js:217:24)
        at new CustomResource (/Users/liamwhite/src/github.com/tetrateio/tetrate/cloud/node_modules/@pulumi/aws/node_modules/@pulumi/pulumi/resource.js:309:9)
        at new Function (/Users/liamwhite/src/github.com/tetrateio/tetrate/cloud/node_modules/@pulumi/lambda/function.ts:467:9)
        at new CallbackFunction (/Users/liamwhite/src/github.com/tetrateio/tetrate/cloud/node_modules/@pulumi/lambda/lambdaMixins.ts:362:9)
        at new CronLambda (/Users/liamwhite/src/github.com/tetrateio/tetrate/cloud/components/aws/lambda/cron.ts:26:28)
        at Object.<anonymous> (/Users/liamwhite/src/github.com/tetrateio/tetrate/cloud/projects/organization/test/managementplane/api/index.ts:61:1)
        at Module._compile (node:internal/modules/cjs/loader:1092:14)
        at Module.m._compile (/Users/liamwhite/src/github.com/tetrateio/tetrate/cloud/projects/organization/test/managementplane/api/node_modules/ts-node/src/index.ts:439:23)
        at Module._extensions..js (node:internal/modules/cjs/loader:1121:10)

So I think if we can get the asset library to handle symlinks, this should just work?

p.s. I'm happy to do this if someone points me in the right direction.

Related, #2980.

I use Yarn workspaces to manage a monorepo containing dozen of pulumi projects and a few library packages.

The key to make this work in my experience is to:

1. Tell Yarn to avoid hoisting your dependencies using nohoist (feel free to improve glob matching for your use case):

{
[...]
  "workspaces": {
    "packages": [
      "lib/*",
      "projects/*"
    ],
    "nohoist": [
      "@yourscope/*/**"
    ]
  }
}

2. Do not declare your monorepo internal dependencies in your package.json (you don't want yarn to mess with it), but use TypeScript project references - Pulumi >3.x works fine with it, even to deploy Lambda functions!

That said, this is a workaround. Making Pulumi work well with Yarn's package hoisting would be awesome!

For Yarn >=v2 you can do this ^^ in the .yarnrc.yml file by adding the following:

nodeLinker: node-modules
nmHoistingLimits: workspaces

Unfortunately, this doesn't work for us as a yarn install now takes 15 minutes (and counting) on the link step. Which, as we use the Pulumi operator means this is not an option for us.

For everyone using yarn 2+ workspaces once #8215 gets into a release, you can do the package.json configuration in my above comment and it should all just work.

is computeCodePaths relevant to this problem?

I stumbled onto this issue trying to get mixins working with pnpm, nx and esbuild

Here's what didn't work for me:

• node-linker=hoisted—deployed fine, but it didn't bundle produced a runtime error (Runtime.ImportModuleError: Error: Cannot find module 'tslib')
• CallbackFunctionArgs.codePathOptions['extraIncludePackages']—deployed fine, but didn't actually include the tslib package

          codePathOptions: {
              extraExcludePackages: ['tslib'],
          },

  

Here's what does work for me:

• node-linker=hoisted in .npmrc

• force tslib to be included in the bundle, using CallbackFunctionArgs.codePathOptions:

  (new aws.cloudwatch.EventRule(/* ... */)).onEvent(
      'hello',
      new aws.lambda.CallbackFunction('hello', {
          runtime: aws.lambda.Runtime.NodeJS18dX,
          codePathOptions: {
              extraIncludePaths: ['../../node_modules/tslib'],
          },
          callback: async (event, context) => {
              console.log('hello from aws lambda!')
          },
      }
  ), {})

       codePathOptions: {
           extraExcludePackages: ['tslib'],
       },

omg I just saw the typo! No surprise tslib was missing from the bundle 🤦🏻

After fixing the typo, it still doesn't work—but at least I get an error message 😂

            codePathOptions: {
                extraIncludePackages: ['tslib'],
            },